The use of deep learning to model flow fields has drawn a lot of attention in recent years. In the context of reduced order modeling (ROM), autoencoders have emerged as alternatives to traditional linear reduction methods, such as proper orthogonal decomposition (POD). Additionally, neural network architectures such as long short-term memory networks and transformers have been integrated into the ROM framework, replacing the Galerkin projection step for temporal modeling in the reduced space. Many of the methods developed perform the reduction and the temporal modeling steps sequentially, i.e., the temporal model is trained once the reduced space is defined [1]. However, recent studies have introduced end-to-end models where the reduction step is implicitly embedded in a temporal model [2], which has the advantage of allowing reduction and temporal modeling to be learned simultaneously, in contrast to having two different models for each step. In light of this, we present an end-to-end spatio-temporal convolutional autoencoder ROM for predicting flow fields in a bed configuration of hot particles. Our model also incorporates physical system knowledge by enforcing boundary conditions, and we evaluate its performance both with and without this added physical information.